Skip to content

AsyncAPI 3.x: an E2E test, NCS driven through Kafka - #1760

Draft
LautaroPetaccio wants to merge 18 commits into
feature/asyncapi-examplesfrom
feature/asyncapi-e2e-kafka
Draft

LautaroPetaccio wants to merge 18 commits into
feature/asyncapi-examplesfrom
feature/asyncapi-e2e-kafka

Conversation

@LautaroPetaccio

@LautaroPetaccio LautaroPetaccio commented Sep 14, 2026 •

Copy link
Copy Markdown
Collaborator

The SUT — NCS over Kafka

The NCS case study re-expressed as a message-driven service: six request topics (ncs.<op>.request), each answered on a reply topic with a result message or, for the inputs the REST version answers with a 400, an error message — exactly what ncs-kafka.yaml (already in core's test resources) describes. The routines are ported from EMB's org.restncs.imp and cited at the top of each file; the rejection rules mirror NcsRest. A Spring Boot application with no web layer: it speaks only Kafka, which is the point.

One deliberate deviation: a result that is NaN or infinite is answered as an error. JSON has no such numbers, and a string where the contract promises a number would be a false fault.

The driver — the reference executeAsyncApiAction

NcsKafkaController owns the broker (testcontainers, apache/kafka:3.8.0, KRaft), creates the twelve topics up front, and implements the hook exactly as #1738's javadoc sketches it:

  • publish with the correlation id in the header the document names — or stamped into the payload at the declared pointer, for documents that say so;
  • the reply consumer is assigned and positioned at the end before publishing, so a fast reply cannot slip past, and reused for the rest of the run;
  • wait until a record carrying the id arrives or replyTimeoutMs runs out; a reply carrying someone else's id is skipped, one carrying none is taken and reported as correlationMatched = false;
  • fill AsyncApiReplyDto and judge nothing.

kafka-clients and testcontainers-kafka appear only in this test module, managed in the root pom.xml as the doc requires. Nothing is added to any shipped jar.

The test

AsyncApiTestBase joins the other bases in e2e-tests-utils: initAndRun, and helpers that read AsyncApiCallResult off the solution. NcsKafkaEMTest runs 300 action evaluations and asserts:

  • every operation was answered;
  • checkTriangle, bessj and remainder reached their result reply;
  • expint and gammq reached both their result and their error — the two operations whose rejected inputs (x < 0; a ≤ 0 || x < 0) the schema does not rule out. bessj's order and remainder's operands are bounded in the schema, so their error replies need invalid data, which the search does not publish yet — worth knowing, and a natural next lever;
  • no NO_REPLY, no PUBLISH_FAILED.

Because the driver is embedded, the SUT is instrumented, so the search covers lines and branches of a message-driven service as well: 662 targets in 26 seconds, identical on two runs.

Not here

The createTests false constraint still stands, so runTestHandlingFlaky is used rather than the compilation variant. Once the test writer exists, this is where its output gets compiled and run too.

@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-e2e-kafka branch from 4b4a66a to 64b5a5a Compare September 14, 2026 11:59
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-e2e-kafka branch from 64b5a5a to e69f386 Compare September 14, 2026 22:19
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-e2e-kafka branch from e69f386 to 8e17ab8 Compare September 15, 2026 23:46
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-e2e-kafka branch from 8e17ab8 to fd3a8b3 Compare September 16, 2026 21:59
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-e2e-kafka branch from fd3a8b3 to 04946ba Compare September 16, 2026 22:01
@LautaroPetaccio
LautaroPetaccio added this pull request to stack #1712 September 20, 2026 22:02
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-e2e-kafka branch from 4c564f5 to df95194 Compare September 20, 2026 22:05
ProblemType.ASYNCAPI, experimental like the other recent types. Main infers
it from SutInfoDto.asyncApiProblem, picks its algorithm key, and stops with
a clear message where the module will go: the sampler and fitness that
complete it are the next changes.

The one decision here is EMConfig.usesDriver(). Every other type reaches
the driver only in white-box mode, or in black-box experiments. An AsyncAPI
service has no universal wire to point at, so the driver holds the broker
connection even when the service is a black box. The places in Main and
Statistics that asked "is there a driver" now ask the config, instead of
each spelling the rule out.
AsyncApiSampler asks the driver where the document is, reads it, builds one
action per publishable message, and samples sequences of them. The first
individuals it hands out publish one message each, one per operation, so
every operation is tried before the search starts combining them.

One sampler for both modes. Where REST and GraphQL fetch the schema
themselves in black-box mode, here the document comes through the driver
either way, as the driver is what holds the connection to the broker.

Seeded test cases are not supported yet, and say so.
Review: what the parser skips was reported to the user and kept for the
final report, but nothing checked it; nor that the single-message
individuals return after a reset, nor the two other ways start-up can
fail. Four tests, and the helper that builds the injector is shared.
Four TODOs, where someone will next look: no AsyncAPI-specific smart
strategy yet, message examples parsed but unused, no seeding format, and a
method shared verbatim with the REST sampler.
RemoteController.executeNewAsyncApiActionAndGetReply PUTs the action to the
driver and reads back an AsyncApiReplyDto, as its RPC counterpart does. It
has a default that throws, so the controllers that never publish need not
say so one by one.

EMConfig gains asyncApiReplyTimeoutMs, experimental, and a constraint:
there is no test writer for AsyncAPI yet, so a run must say
--createTests false rather than search for an hour and fail at the end.
The two tests that already parsed --problemType ASYNCAPI say so now.

Two experimental fault categories: a promised reply that never arrives,
and a reply matching none of the messages the contract declares.
AsyncApiBlackBoxFitness publishes each message of a test through the
driver and turns what comes back into targets: for every operation, what
publishing to it was seen to do, and, when a reply came back, which of the
messages the contract declares for the reply it was. A contract listing a
result and an error thus gives the search two things to reach, which is
the AsyncAPI analogue of REST's (status x endpoint).

AsyncApiReplyClassifier is what recognises a reply: a structural match
against each declared payload schema, reading the parts of JSON Schema
that tell one message from another and giving the benefit of the doubt on
the rest. The most specific match wins.

AsyncApiModule binds it all, with the driver bound unconditionally, and
Main uses it in place of the message it showed until now. One test runs a
whole MIO search against a stand-in for the NCS service and sees both
declared replies of an operation covered.
Review: with room for exactly two messages, a one-message test was
mutated by removal, leaving a test that publishes nothing; and a test
could never grow to the maximum the user allowed, only to one less. Both
bounds are now stated as such, and AsyncApiStructureMutatorTest holds
them.

Seeding test cases is refused at start-up, like writing them, instead of
failing inside the sampler. The whole-search suite is named after the
class it exercises, AsyncApiModule, and the three suites that build the
injector share how they do it.
Review: the ':' joining a target id and the '-' inside a correlation id
were spelled out where used; they are constants now, and the two kinds of
target id are built in one place each. The remote controller named its
queryFromDatabase parameter four times over, once in the new call and
three in the ones it mirrors; it is a constant now. The fake driver's one
field comes before its companion.
…aults

Review findings, in order of what they cost a run.

The address a message is published to ignored protocol bindings. A Kafka
channel usually names no address and carries its topic in its binding --
microcks.yaml, already a test resource here, is exactly that shape -- so
messages went to the channel's name and reached nobody. The parser's
effectiveAddress() exists for this and was never called; when a channel
names no address at all, the binding's topic is now taken as the one
destination the document actually gives. An address left holding
{placeholders} is warned about rather than published to silently.

Deciding whether a number is an integer asked Jackson for a BigDecimal,
which throws for a value that overflows a double. Nothing between the
classifier and the search loop catches it, so one odd reply ended the run.

Three ways a valid reply was reported as a fault: a body that is empty or
not JSON, a const or enum written 1.0 against a reply carrying 1, and a
type or combinator this cannot read. A reply with nothing to match against
is now no finding at all, numbers compare by value, and what cannot be read
rejects nothing. A $ref that points into a schema is followed instead of
matching everything, which had quietly disabled the undeclared-reply oracle
for any operation written that way, and specificity counts through the
combinators so the more specific message wins.

Faults ignored isEnabledFaultCategory, so two experimental oracles fired on
a default run and --disabledOracleCodes did nothing. They are recorded on
the action result now, which is where the reports count faults from; before,
a run covered fault targets while every report said zero.

AsyncApiCallResult was the only action result not overriding matchedType.

An inferred problem type no longer dies on the test-generation constraint:
when the driver reports an AsyncAPI service, createTests is turned off with
a warning instead of throwing after the SUT has already been started.

The class is AsyncApiFitness: it is bound as the only fitness and reads
white-box coverage, so BlackBox in its name said something untrue.
The new remote-controller call was a copy of the RPC one; both now share
the body that PUTs an action and reads the reply back.

The structure mutator is a third copy of logic REST and RPC already share,
marked with a TODO the way the GraphQL one marks the same debt.

The reply-timeout description is published verbatim into options.md, where
every neighbour is one clause; the reasoning behind the option belongs in
the pull request, not in the user's terminal.
The guards that keep a valid reply from being reported as a fault had no
tests, which is why they could be wrong. Now covered: a reply with no body,
an operation whose reply declares no message, a number too large to be a
decimal, a const compared by value, a reference into a schema, and the most
specific match found through a combinator.

Also covered: the topic a protocol binding names, the configured reply
timeout, and that faults stay quiet until experimental oracles are asked
for -- each of which would otherwise pass with the code deleted.

The structure mutator's tracking is exercised with a real specification
rather than null, so the add/remove bookkeeping the archive reads is
checked. The module test asserts the bindings Main resolves, since this
module deliberately inherits none. The sampler suite resets the static gene
cache its siblings all reset.
The reply DTO no longer uses primitives, so a field the driver never set
arrives as null rather than as a default. Kotlin maps those to platform
types, so reading them as before would still compile and throw at run time.

Each is now read deliberately. Whether the message was published has no
safe default, so an absent answer is treated as a failure and said so in
the error message, apart from a driver that answered false. Whether a reply
was expected or arrived is read as not having happened when unset, which is
what a fire-and-forget driver leaves blank.

Correlation is the one that changes behaviour: a driver that does not track
it at all reported false, which reads as the service having failed to echo
the id back. That is a claim about the SUT the driver never made, so it is
now recorded only when the driver actually looked.
Turning the experimental oracles off used to cost coverage rather than only
reporting. A reply matching none of the declared messages registered nothing
beyond the gated fault call, so it was indistinguishable from a recognised one
and could be dropped when the solution was minimised. It now covers a target of
its own, registered before the fault and independent of whether it is enabled,
which is how a 500 is already handled for REST.

Also in this round:

- the correlation id prefix no longer comes from the seeded generator, which
  made a run repeated under the same seed reuse the very ids it exists to tell
  apart
- an outcome that cannot reach target handling now says so by throwing rather
  than by a comment, and the branch stays exhaustive so that a newly added
  outcome fails to compile instead of falling through
- why there is no separate black-box fitness, and why the reply classifier is
  written by hand rather than delegated to the validator already on the
  classpath, which reads draft-04 and so does not know const
A document's message examples are what its author knows the service
accepts. A service that silently drops what it does not recognise may
never answer a sampled payload, so with --probAsyncApiExamples the first
example of a message is offered as a whole value beside the schema-derived
genes, at that probability. Off by default, like every new feature.

It reuses what REST already does with a schema's 'example': the example is
put on the message's own copy of the payload schema, and the gene builder
turns it into the same choice it offers REST. Only the first example is
used: the OpenAPI parser the genes go through reads the schemas as 3.0,
which drops the plural 'examples'. Scalars are left alone, as the builder
would complain about 'example' on them. A headers example loses the field
the correlation id is stamped into, as the headers gene did before it.
The first cut wrote one example into the message's copy of its payload
schema and let the gene builder find it there. That path crosses the
OpenAPI parser, which keeps only the first of several, drops the name an
example may carry, and warns about an example on a scalar. So only one
example was offered, never by name, and never for a scalar payload.

The gene builder already takes examples the other way, as a list of values
with optional names, for a REST parameter or request body. Its entry point
for a bare schema simply did not expose that parameter. It does now, with
an empty default so that no other caller changes, and the AsyncAPI builder
passes every example through it: all of them are offered, a name survives
to where the sampler's named-example pass can read it, and a scalar
payload gets the same choice an object does.

One trap came with it. The builder caches what it builds by schema text
alone, so two messages sharing a schema but declaring different examples
would have been handed the same gene. A call that brings examples now
neither reads nor writes that cache.
The first search against a real broker. The SUT is the NCS case study
re-expressed as a Kafka service: six request topics answered on six reply
topics with a result or an error, as ncs-kafka.yaml describes. It speaks
only Kafka; there is no HTTP endpoint. Its routines are ported from EMB's
NCS and cited as such.

The driver owns the broker, a testcontainers apache/kafka, and is the
reference implementation of SutController.executeAsyncApiAction: publish
with the correlation id where the document says it goes, wait for the
reply that carries it back, report and do not judge. kafka-clients and
testcontainers-kafka are used only here, in the test tree, and are managed
in the root pom like everything else.

The test asserts that every operation answers and that expint and gammq
each reach both their declared replies. Those two are the operations whose
rejected inputs the schema does not rule out; bessj's and remainder's
bounds are in the schema, so their errors need invalid data, which the
search does not publish yet. 662 targets in 26 seconds, on two runs.
Neither side of the experimental oracle gate was exercised against a live
broker. A second search now runs with the oracles on and a reply deadline no
round trip can meet, so every message goes unanswered and the fault is
reported, while the existing search asserts that a service which answers
correctly has nothing reported against it.

Nothing in the service is broken on purpose for this. The driver ignores
replies whose correlation id does not match the request it is waiting on, so
the late answers piling up on the reply topics cannot be taken for fresh ones.
@arcuri82
arcuri82 force-pushed the feature/asyncapi-e2e-kafka branch from df95194 to 5609333 Compare September 22, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant